Zapformer preview - #2082

Hi @danpovey, first, thanks a lot to you and your team for sharing this new architecture. I have a question here, have you tried CR-CTC(+pruned transducer) with this? Does this new architecture constitute similar improvements as Zipformer with CR-CTC over no CR-CTC version?

Hi @danpovey, first, thanks a lot to you and your team for sharing this new architecture. I have a question here, have you tried CR-CTC(+pruned transducer) with this? Does this new architecture constitute similar improvements as Zipformer with CR-CTC over no CR-CTC version?

No; for reasons that are unclear to me, the CR-CTC loss does not seem to help in this setup and with this model.

For non streaming, I get indeed much better results:

python ./zapformer/decode.py     --epoch 30     --avg 3     --exp-dir ./zapformer/exp     --max-duration 1000     --decoding-method greedy_search --causal true --chunk-size 32 --left-context-frames 256
greedy_search	3.56	best for dev-clean
greedy_search	9.85	best for dev-other
greedy_search	3.79	best for test-clean
greedy_search	9.82	best for test-other

If I try with avg 15 to compare, I get similar results (~3.5% - 9.9%)

For the streaming script, with only 3 epochs it's even worse than 15:

python ./zapformer/streaming_decode.py   --epoch 30   --avg 3  --causal 1   --chunk-size 32   --left-context-frames 256   --exp-dir ./zapformer/exp   --decoding-method greedy_search   --num-decode-streams 1000
greedy_search	31.47	best for dev-clean
greedy_search	30.95	best for test-clean
greedy_search	45.98	best for dev-other
greedy_search	46.06	best for test-other

so there may be something wrong in the streaming decoding... It could also be due to some small changes I had to do to make the onnx export run in the first place. If I have time I will test your updated branch and report again.

Thanks!

Thanks for reporting this, I'll check with Kangwei. Let us know what you had to do to make the onnx work!

For non streaming, I get indeed much better results:

python ./zapformer/decode.py     --epoch 30     --avg 3     --exp-dir ./zapformer/exp     --max-duration 1000     --decoding-method greedy_search --causal true --chunk-size 32 --left-context-frames 256
greedy_search	3.56	best for dev-clean
greedy_search	9.85	best for dev-other
greedy_search	3.79	best for test-clean
greedy_search	9.82	best for test-other

If I try with avg 15 to compare, I get similar results (~3.5% - 9.9%)

For the streaming script, with only 3 epochs it's even worse than 15:

python ./zapformer/streaming_decode.py   --epoch 30   --avg 3  --causal 1   --chunk-size 32   --left-context-frames 256   --exp-dir ./zapformer/exp   --decoding-method greedy_search   --num-decode-streams 1000
greedy_search	31.47	best for dev-clean
greedy_search	30.95	best for test-clean
greedy_search	45.98	best for dev-other
greedy_search	46.06	best for test-other

so there may be something wrong in the streaming decoding... It could also be due to some small changes I had to do to make the onnx export run in the first place. If I have time I will test your updated branch and report again.

Thanks!

Can you share your training command, my model trained with num-real-epochs=30, and decoded with chunk-size=32 left-context-frames=128, the WER on test-clean is 2.38, much lower than yours (is 3.79).

For non streaming, I get indeed much better results:

python ./zapformer/decode.py     --epoch 30     --avg 3     --exp-dir ./zapformer/exp     --max-duration 1000     --decoding-method greedy_search --causal true --chunk-size 32 --left-context-frames 256
greedy_search	3.56	best for dev-clean
greedy_search	9.85	best for dev-other
greedy_search	3.79	best for test-clean
greedy_search	9.82	best for test-other

If I try with avg 15 to compare, I get similar results (~3.5% - 9.9%)
For the streaming script, with only 3 epochs it's even worse than 15:

python ./zapformer/streaming_decode.py   --epoch 30   --avg 3  --causal 1   --chunk-size 32   --left-context-frames 256   --exp-dir ./zapformer/exp   --decoding-method greedy_search   --num-decode-streams 1000
greedy_search	31.47	best for dev-clean
greedy_search	30.95	best for test-clean
greedy_search	45.98	best for dev-other
greedy_search	46.06	best for test-other

so there may be something wrong in the streaming decoding... It could also be due to some small changes I had to do to make the onnx export run in the first place. If I have time I will test your updated branch and report again.
Thanks!

Can you share your training command, my model trained with num-real-epochs=30, and decoded with chunk-size=32 left-context-frames=128, the WER on test-clean is 2.38, much lower than yours (is 3.79).

See for the training command:
#2082 (comment)
but as mentioned, it was on an older commit. Probably more recent ones give better results.

./zapformer/train.py \
  --world-size 1 \
  --num-real-epochs 30 \
  --start-epoch 1  \
  --use-bf16 1 \
  --exp-dir zapformer/exp \
  --causal 1 \
  --full-libri 1 \
  --max-duration 1000

Hi @danpovey , thank you for sharing Zapformer, If training Zapformer with other datasets, how should the --batches-per-epoch parameter be adjusted?

Hi @danpovey , thank you for sharing Zapformer, If training Zapformer with other datasets, how should the --batches-per-epoch parameter be adjusted?

batcher-per-epoch is not an important configuration value. It is supposed to be the number of batches in the 1st epoch, but it will make almost no difference if you don't set it, for typical learning rate schedules, because it automatically figures out the real number of batches per epoch by the second epoch.

Hi @danpovey , thank you for sharing Zapformer, If training Zapformer with other datasets, how should the --batches-per-epoch parameter be adjusted?

batcher-per-epoch is not an important configuration value. It is supposed to be the number of batches in the 1st epoch, but it will make almost no difference if you don't set it, for typical learning rate schedules, because it automatically figures out the real number of batches per epoch by the second epoch.

When training on a large dataset, such as 100Khours, typically we don’t need to train for as many as 40 epochs. Should I need to adjust the --max-copies or some other parameter?

Yes, you would probably want a much smaller --max-copies, e.g. 2 or 1... it's there to prevent overfitting from a large number of epochs, if the num epochs is small it shouldn't be necessary.